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ABSTRACT 



In large-scale assessments, some amount of background 
material is generally collected. Background questionnaires in the National 
Assessment of Educational Progress are the subject of this report. Most of 
the questions in NAEP background questionnaires have asked about course 
taking patterns and course content, but it is argued that the material 
gathered at present is of dubious quality and that information that is more 
extensive and valid could be collected. The questions about courses students 
have taken are often unnecessary or meaningless, and the coverage of courses 
is absent or incomplete. Information about advanced placement courses is 
demonstrably invalid, and information about other courses is often vague. 
Specific solutions, aimed at greater accuracy and detail, are proposed for 
these problems. A second major problem area is in the language that is used 
on the background questionnaires for fourth grade, which may be too difficult 
for the average student. Field testing these proposed changes is discussed. 

An attachment shows the percentage of 12th graders who report that they have 
taken advanced placement courses. (SLD) 



******************************************************************************** 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 




TM !b~z 



O 

r- 

i— ( 
8 



NATIONAL CENTER FOR EDUCATION STATISTICS 



Working Paper Series 



Innovative Solutions to 
Intractable Large Scale Assessment 

( Problem 2: Background Questionnaires) 



Working Paper No. 97-32 



October 1997 




♦ . • * 






r 



j o Jt? - 









U.S. Department of Education 

Office of EducationalResearch and Improvement 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Imorovement 

EDUCATIONAL RESOURCES INFORMATION 
. CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 

• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 





BEST COPY AVAILABLE 





Innovative Solutions to 
Intractable Large Scale Assessment 

( Problem 2: Background Questionnaires) 



Working Paper No. 97-32 



October 1997 



Contact: Steven Gorman 

Assessment Group 
(202)219-1937 

e-mail: steven_gorman@ed.gov 



U.S. Department of Education 

Richard W. Riley 

Secretary 

Office of Educational Research and Improvement 

Ricky T. Takai 

Acting Assistant Secretary 

National Center for Education Statistics 
Pascal D. Forgione, Jr. 

Commissioner 

Assessment Group 
Gary W. Phillips 
Associate Commissioner 



The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, 
and reporting data related to education in the United States and other nations. It fulfills a congressional 
mandate to collect, collate, analyze, and report full and complete statistics on the condition of education 
in the United States; conduct and publish reports and specialized analyses of the meaning and significance 
of such statistics; assist state and local education agencies in improving their statistical systems; and review 
and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, 
complete, and accurate irtdicators of education status and trends; and report timely, useful, and high quality 
data to the U.S. Department of Education, the Congress, the states, other education policymakers, 
practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a 
variety of audiences. You, as our customer, are the best judge of our success in communicating 
information effectively. If you have any comments or suggestions about this or any other NCES product 
or report, we would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 
Office of Educational Research and Improvement 
U.S. Department of Education 
555 New Jersey Avenue, NW 
Washington, DC 20208 



Suggested Citation 

U.S. Department of Education. National Center for Education Statistics. Innovative Solutions to Intractable Large Scale 
Assessment (Problem 2: Background Questionnaires), Working Paper No. 97-32, by Richard G. Niemi. Project Officer, 
Steven Gorman. Washington, D.C.: 1997. 



October 1997 



Foreword 



Each year a large number of written documents are generated by NCES staff and 
individuals commissioned by NCES which provide preliminary analyses of survey results and 
address technical, methodological, and evaluation issues. Even though they are not formally 
published, these documents reflect a tremendous amount of unique expertise, knowledge, and 
experience. 

The Working Paper Series was created in order to preserve the information contained 
in these documents and to promote the sharing of valuable work experience and knowledge. 
However, these documents were prepared under different formats and did not undergo 
vigorous NCES publication review and editing prior to their inclusion in the series. 
Consequently, we encourage users of the series to consult the individual authors for citations. 

To receive information about submitting manuscripts or obtaining copies of the series, 
please contact Ruth R. Harris at (202) 219-1831 or U.S. Department of Education, Office of 
Educational Research and Improvement, National Center for Education Statistics, 555 New 
Jersey Ave., N.W., Room 400, Washington, D.C. 20208-5654. 



Samuel S. Peng 
Acting Director 

Statistical Standards and Services Group 



Innovative Solutions to Intractable Large Scale Assessment 
(Problem 2: Background Questionnaires) 



Prepared by: 

Richard G. Niemi 
Rochester, New York 



Prepared for: 

U.S. Department of Education 
Office of Educational Research and Development 
National Center for Education Statistics 



October 1997 



Table of Contents 









» 



I 



» 



9 



9 



I 



9 



9 



Problem 1: Quality/Specificity of Information about Courses Students Have Taken 1 

(1) Students Are Asked Unnecessary or Meaningless Questions 2 

(2) Coverage of Courses Is Absent or Incomplete 3 

(3) There Are Un-noted/unexplained Shifts in Which Courses Are to Be 

Considered 4 

(4) Information about AP Courses Is Demonstrably Invalid 4 

(5) Information about Courses Taken Is Vague and (Probably ) Much Less 

Accurate Than Necessary, Especially in the Area of Civics 4 

Solutions: 

(1) Make Clear, Conscious Decisions about Who Should Be Asked Which 
Questions; Alert Students to the Basis for Answering Questions (If Not 



Obvious); Introduce Skip Patterns or “Stop” Instructions to Existing 
Questionnaires 5 

(2) When Inquiring about Courses, Be Sure That Coverage Is Complete; 

Possibly Eliminate the Phrase “Do You Expect to Take” a Course 8 

(3) Decide Which Course(s) to Ask About; If There Is a Change at Some 

Point in the Questionnaire, Alert Students to the Change 8 

(4) Refer to AP Courses by Their Full and Correct Title 8 

(5) Use a Method of Gathering Course-taking Information from Students 
That Refers to Specific Courses Offered at the School in Which the 

Student Is Being Tested 9 

Problem 2: Level of Language/Understanding Expected of Fourth Graders 16 

Solution: Simplify the Vocabulary and the Phrasing Used 18 

Field-Testing of New Methods 1 9 

Attachment 1: AP Courses by Grade and High School Program (1988) 21 

vii 



O 

ERIC 



7 



INNOVATIVE SOLUTIONS TO INTRACTABLE LARGE SCALE ASSESSMENT 

(Problem 2: Background Questionnaires) 



While the primary purpose of NAEP is to “describe how well students are performing, but not to 
explain why” (NAGB Policy Statement on Redesigning NAEP, 1 996, 4), some amount of 
background information has always been collected. A portion of the background material has 
been subject specific. It is these “subject questionnaires” that are my concern in this report. In 
these questionnaires, most of the questions have asked about course-taking patterns and course 
content. It is my contention that the information that is presently gathered (especially about 
which courses students have taken) is of dubious quality and that information that is both more 
extensive and valid could be collected. 

I organize the report around two problems. The first is the quality and specificity of information 
about courses students have taken. The second is the level of language and understanding 
expected of 4th graders. In each case, I first indicate what the problem is and then suggest one or 
more solutions. In the final section, I discuss ways in which the proposed new methods might be 
field-tested. 



Problem 1: Quality/Specificity of Information about Courses Students Have Taken 

In this section my main concern is the subject questionnaires used at the 12th grade, especially 
for gathering information about social studies course work (though I have some comments on the 
writing and reading questionnaires). By 12th grade, students within a single school may have 
taken a number of different social studies offerings (world history, U.S. history, geography, 
American government, comparative government, participation in government, economics, 
psychology, etc.) at two or three different levels, and, across schools, the courses are known by 
an extraordinarily diverse set of names. Nonetheless, students have simply been asked whether 
they had “a geography course” or a “United States history course,” and whether they have 
“studied American government or civics.” It is both appropriate and feasible to collect much 
more refined information. 1 

I divide this overall problem into five parts. Parts 1-4 are grouped together and placed first 
because they call for only minor changes to the existing questionnaires. Part 5 would require a 
significant change in the way information is gathered; however, the solution suggested in Part 5 
also holds the greatest potential for improvement in extensiveness and quality of information 
about course-taking patterns in all subject areas. 



'I have not examined questionnaires in mathematics and science, but a similar critique almost surely applies. 



(1) STUDENTS ARE ASKED UNNECESSARY OR MEANINGLESS QUESTIONS. 



A common practice in survey instruments is to use skip patterns and skip instructions so that 
respondents are not asked “inappropriate” questions. A simple example is a question on length of 
marriage; it is typically not asked of those who are currently unmarried. Indeed, asking 
questions that “do not apply" is potentially confusing, invites unreliable and invalid responses, 
and, in the usual situation, wastes valuable respondent time. 

This practice is not followed in the NAEP questionnaires. The problem arises for students who 
have not had any history/civics/geography/etc. courses and yet are asked about course content 
and procedures. In some instances, this may not be a problem. We can safely assume, for 
example, that almost all high school seniors have had a course on U.S. history, but only three- 
quarters have had a course in American government and politics, and only one-quarter a course 
in geography. 2 

Examples: 

1994 Reading, grades 8 and 12: Q. 19 asks students whether they have access to a school 
or public library. Even if they say “no,” they are expected to answer Q’s 20-24 about using 
the school or public library. 

1994 Geography, grade 12: Q’s 5-6 ask students whether they have taken a course in world 
or in U.S. geography although some will have indicated in Q’s 1-4 that they have had no 
geography course in grades 9-12. Q’s 5-6 could apply to courses at any level of school 
(e.g., 8th grade), but that is not made clear and the context established by the first four 
questions is the high school. 

1994 Geography, grade 12: Q’s 8-14 inquire about topics such as geography homework 
and are asked of all students. Of course, students could have studied geography in other 
classes, but there is no indication in the questionnaire about how students should answer 
these questions if they have had no geography course or no formal study of geography at 
all. 

1998 Civics (pre-test, grade 12): Q’s 5-13 ask students whether they have studied various 
topics related to government even if they have just indicated that they have not studied 
civics or government in grades 9-12. Q’s 14-25 ask all students about how they study 
civics/ government, even if they have just indicated that they have not studied any of the 



2 The figures for 1994 were: history (1.00 Carnegie credits)-- 95%; American government and politics (.50 
credits)-- 78%; geography (.50 credits)~25%. Stanley Legum, et al., The 1994 High School Transcript Study 
Tabulations: Comparative Data on Credits Earned and Demographics for 1994, 1990, 1987, and 1982 High School 
Graduates (Washington, D.C.: National Center for Education Statistics, 1997), A- 199. 



3 



listed topics. Q. 26 asks all students about their most recent civics/govemment class. And 
® Q. 28 asks all students whether they have a textbook on civics/govemment. 

In the civics example, one might argue that there is a more fundamental problem. That is, 
students may misinterpret what the study of civics/govemment is about and indicate that they 
have not studied it when, in fact, almost all students have studied government— e.g., in a history 

• class. We have more to say about this in Part 5. But for the moment, even the fact that most 
students have studied government someplace does not make Q. 26 appropriate for all students, 
since it inquires about their most recent civics/govemment class. 

• (2) COVERAGE OF COURSES IS ABSENT OR INCOMPLETE. 

On the pre-test for the 1998 Civics Assessment, students are not asked about civics courses at all. 
They are asked “In what grades(s) have you studied civics or government [9-12]? For more 
discussion of this point, see Part 5. 

I 

On the 1994 History Assessment, students were asked whether they had taken a United States 
history course in 9th-llth grades (Q’s 1-3). They were then asked, “Are you taking a United 
States history course now?” Note that if there existed a one-semester course that a student took 
during the fall of the senior year, there would be no place to indicate having taken that course, 
ft While the “basic” U.S. history course is almost always a year long, more specialized, one- 

semester courses (designed precisely for the senior year) exist in some schools throughout the 
country. 3 

On the 1994 Geography Assessment, students were asked “did you take or do you expect to take 
ft a geography course in the following grades [9-12]?” (italics added) For students given the 

Assessment in February and March, the italicized portion is irrelevant; if the same is true in 
January, these italicized words could be deleted. Note also that there could be some, perhaps 
slight, confusion on the part of students currently enrolled in a geography course. These students 
did not (already) take the course but neither do they expect to take it. 

» 

In the pre-test for the 1998 Civics Assessment, students are asked only whether they “have 
studied" the U.S. Constitution, Congress, etc.— i.e., the responses are only “yes” and “no.” Unlike 
in other subject questionnaires, students are not asked to indicate relative amounts of study. 

ft 



3 In one of the schools from which I gathered course information, there is a one-semester course called 
“Critical Issues/World at War, which would seem to be precisely this kind of course. 
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(3) THERE ARE UN-NOTED/UNEXPLAINED SHIFTS IN WHICH COURSES ARE TO BE 
CONSIDERED. 



In the 1994 History Assessment, Q’s 1-9 are exclusively about U.S. history. Then, without 
explanation or warning, Q’s 10-24 ask about "history or social studies.” There are three 
concerns: a) Would most analysts using the U.S. History NAEP really want student responses 
about history and social studies? b) Would such analysts want student responses about the study 
of any history? c) Do students pick up on this shift and respond to the question as asked? 

From an analytical point of view, I see (a) and (b) as significant problems. Given that this is a 
U.S. History Assessment, I would presumably be most interested in information about U.S. 
history courses. But that is not what the question gives me. Point (c) is a problem even if I am 
satisfied with information about history and social studies because I am not sure how students are 
actually answering the question. 

In the 1994 Geography Assessment, almost all of the questions are exclusively about geography. 
Q. 7, however, inquires about “geography or social studies.” 

(4) INFORMATION ABOUT AP COURSES IS DEMONSTRABLY INVALID. 

While working with the 1988 Civics Assessment, I found that students with self-reported grades 
of D were as likely to say that they took an AP course as students with mostly A’s; students in 
general and vocational/technical tracks reported taking an AP course as often as those in 
academic tracks (see Attachment 1). Obviously there is some confusion about what is meant by 
AP courses. 

The same question about AP courses was used in the 1994 History, Geography, and Reading 
Assessments. At my suggestion, a new version of the question is being tested (see Solutions 
below). 



(5) INFORMATION ABOUT COURSES TAKEN IS VAGUE AND (PROBABLY ) MUCH 
LESS ACCURATE THAN NECESSARY, ESPECIALLY IN THE AREA OF CIVICS. 

If the intent of the subject questionnaires is to find out about course work, Parts 1-4 indicate 
considerable vagueness and room for error/misinterpretation on the part of students. But there 
are additional points as well. For example, even when students are asked about particular 
courses (e.g., U.S. history, 9th grade), it is impossible to judge the level of course work or to 
make an independent judgment about the probable focus of courses taken. 

The problem of ambiguity is heightened in civics. The difficulty is that civics/govemment 
material may be taught in separate courses or integrated into other courses, especially history. 



This makes it inappropriate to ask the course question as it is done in history and geography: 
“Did you take a civics or American government course in grades 9-12.” On the other hand, the 
question that is used— "have you studied civics or government”— leaves it up to the student to 
decide what material falls under that rubric. 4 This ambiguity about what constitutes the study of 
civics/ government must, in turn, affect in unknown ways the responses of students to questions 
about classroom activities. 

As a result of allowing students to judge what is and is not a civics/govemment course, one has 
no indication of whether or not a student has taken a course wholly devoted to the subject. A 
student can indicate that he or she studied civics or government, but there is nothing to indicate 
the course(s) in which that study occurred. 



Solutions: 

(1) MAKE CLEAR, CONSCIOUS DECISIONS ABOUT WHO SHOULD BE ASKED 
WHICH QUESTIONS; ALERT STUDENTS TO THE BASIS FOR ANSWERING 
QUESTIONS (IF NOT OBVIOUS); INTRODUCE SKIP PATTERNS OR "STOP” 
INSTRUCTIONS TO EXISTING QUESTIONNAIRES. 

First, some person or group-presumably the committee charged with overseeing the preparation 
of the assessment in that subject— must decide which questions should be asked of all students 
and which should be asked only of students who have had relevant course work. This will not 
always be a simple matter. 

Example: 

1994 Geography, grade 12: Here one would have to decide whether to ask Q’s 8-28 of all 
students. My own opinion: 

Q. 16 (how much you like studying geography) can sensibly be asked of all students (and 
should be). 

Q. 9 (how much geography homework you have) is inappropriate for students without 
courses. 

Q’s 8, 10-15, 17-28. These represent a more difficult decision. One might argue that 
everyone should be asked these questions because all students study at least a minimal 
amount of geography in some social studies class. I find that assertion somewhat dubious, 

4 For example, a student might wonder whether “Global Studies,” taught in many New York State schools, 
involves the study of civics/govemment. (According to one school catalogue, “the concept of interdependence and 
the accelerated frequency of interactions among nations and people is a primary focus of this course.”) 



but the point is that this decision should be made very consciously and by people 
knowledgeable about the subject matter. 

Second, if the basis for answering the questions is not obvious, instructions on the questionnaire 
should alert students to the basis for answering the questions. 

Example: 

1994 Geography, grade 12: A statement before Q. 8 might read: 

ALL STUDENTS SHOULD ANSWER Q’S 8, 10-15. IF YOU HAVE NOT HAD ANY GEOGRAPHY 
COURSE IN GRADES 9-12, ANSWER ON THE BASIS OF GEOGRAPHY LESSONS THAT OCCUR IN 
OTHER COURSES SUCH AS HISTORY AND GOVERNMENT. 

Note: I find this instruction somewhat awkward, which is partly the point. If it is not easy 
for students to understand how they are supposed to answer the questions, perhaps they 
should not be asked. 

Third, the layout of the questionnaire must be modified so that skip instructions are as simple as 
possible. Among other things, this might result in rearrangements of the questions. In the 
geography example, Q. 1 6 and Q. 29 (about AP History) might be grouped together if they, but 
not others, were to be asked of all students. 

A simple example of how skip instructions might look follows: 
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Questions 1-3. Did you take a geography course in the following grades? Fill in one oval for 
each question. 





Yes 


No 


I Don’t Know 


1. 9th grade 


O 


O 


O 


2. 10th grade 


O 


O 


O 


3. 11th grade 


o 


O 


O 


4. 12th grade (first semester) 


o 


O 


O 


5. Are you taking a geography course now? 
O Yes 
O No 




If you have taken or are now 




If you have never taken a geography 


taking a geography course, 
continue on with Question 6. 




course and are not taking one now, 
SKIP to Question 16. 



I 



6. Question 6 goes here. 



► ERIC 



14 



(2) WHEN INQUIRING ABOUT COURSES, BE SURE THAT COVERAGE IS COMPLETE; 
POSSIBLY ELIMINATE THE PHRASE “DO YOU EXPECT TO TAKE” A COURSE. 



Information about civics courses (as opposed to the content of civics lessons) cannot easily be 
gathered using the current questionnaire, even with reasonable modifications. For a suggested 
solution to this problem, see Part 5. 

Complete course coverage requires special care in constructing the questions, paying special 
attention to the senior year. The greatest practical difficulty may be that some schools start new 
courses after the January assessment is given. If so, the phrase “do you expect to take” a course 
must be maintained. The suggested solution in Part 5 might circumvent this problem. 



(3) DECIDE WHICH COURSE(S) TO ASK ABOUT; IF THERE IS A CHANGE AT SOME 
POINT IN THE QUESTIONNAIRE, ALERT STUDENTS TO THE CHANGE. 

Someone— presumably the committee charged with overseeing the preparation of the assessment 
in that subject— must decide exactly which course(s) should be the subject of inquiry. My own 
feeling is that it makes most sense to stick to one subject— not, for example, to include social 
studies in one and only one question, as in the 1994 Geography Assessment. 

If the basis for answering the questions changes, instructions on the questionnaire should alert 
students to that change. 



(4) REFER TO AP COURSES BY THEIR FULL AND CORRECT TITLE. 

Previously the question was: 

“Have you taken or are you currently taking an Advanced Placement course in 
[subject]?” 

In the pre-test for the 1998 Civics Assessment, the question was changed to: 

“Have you taken or are you currently taking the College Board Advanced Placement 
course in United States Government and Politics ?” 

Reference to the College Board and to the course by its full title may eliminate gross 
misunderstanding and misreporting by students. I assume that tests will be made to try to 
validate the responses to the new wording. 



(5) USE A METHOD OF GATHERING COURSE-TAKING INFORMATION FROM 
STUDENTS THAT REFERS TO SPECIFIC COURSES OFFERED AT THE SCHOOL IN 
WHICH THE STUDENT IS BEING TESTED. 



Section (a) 

Fundamentally, asking students about high school courses they have had is difficult because of 
the great array of such courses when viewed from a national perspective. In the social studies 
area alone, the 1994 Transcript Study lists over 300 distinct course titles. Even in the highly 
structured field of mathematics there are well over 100. 

In a given school, however, the number of courses in any given subject is relatively small. Thus, 
it may be possible to ask students about the courses they have had using the exact course titles in 
their school. The advantages of this approach are many: 

Much more specific data about course-taking can be gathered. 

More detailed data— e.g., about course lengths— could easily be coded. 

It is easier to ask about courses in multiple subjects (e.g., both history and civics). 

It is easier to ask about courses yet to be taken/currently being taken. 

Data about courses taken are likely to be more reliability and more valid. 

Data about course content and methods are likely to be more reliable/valid. 

It is possible to gather information about civics/govemment courses as opposed to 
information solely about the teaching of civics material. 

There are some potential problems/disadvantages as well. However, let me first explain how 
such a procedure might work. 

One would begin by gathering lists of available courses (in the relevant subjects) from the high 
schools in the NAEP sample. The first page of the subject questionnaire would then be made 
school-specific. Consider the following example. I gathered course information on all subjects 
from several school districts in upstate New York. The following are lists of courses in social 
studies: 



(Small Suburban) High School 

Social Studies (Basic and regular) 9 
Social Studies (Basic and regular) 10 
Social Studies (Basic and regular) 1 1 
AP American History 
Law and Government 
AP Government and Politics 



(Small Town/consolidated) High School 
Global Studies I 
Global Studies II 

United States History and Government 
Economics/Participation in Government 
Critical Issues/World at War 
Criminal Justice 

Advanced Placement History/American Voices 



(Large Suburban) High School District 

American History 

U.S. History and Government 

Participation in Government Internship 

Participation in Government 

Contemporary Issues 

AP American History 

AP U.S. Government & Politics 



(Urban) High School District 
Global Studies I (regular and honors) 

Global Studies 10 

Global Studies II (regular/honors/regents) 
U.S. History/Gov’t (regular./honors/regents) 
Participation in Government 
Economics 
AP American History 
AP European History 



Below I have designed two possible formats for asking students about the courses they have 
taken. For illustrative purposes, I use the courses in the large suburban high school. 



Format 1 : 



Questions la-lg. Have you taken or are you taking any of the following courses taught at 
NAME High School? Fill in one oval for each question. (If you transferred from another 
school, fill in circles for the courses that are closest to those named.) 

I Already I Am Taking Never took it, 





Took It 


It Now 


Not taking 


la. American History 


O 


O 


O 


lb. U.S. History and Government 


O 


O 


O 


lc. Participation in Government Internship 


O 


O 


O 


Id. Participation in Government 


O 


O 


O 


le. Contemporary Issues 


O 


O 


O 


If. A.P. American History 


O 


O 


O 


lg. A.P. U.S. Government & Politics 


O 


O 


O 



Format 2: 



Questions la-lg. Have you taken or are you taking any of the following courses taught at 
NAME High School? Fill in one oval for each question. (If you transferred from another 
school, fill in circles for the courses that are closest to those named.) 



Ninth 

Grade 



la. American History O 

lb. U.S. History and Government O 

lc. Participation in Government Internship O 

l d. Participation in Government O 

le. Contemporary Issues O 

l f. A.P. American History O 

lg A.P. U.S. Government & Politics O 



Tenth Eleventh Twelfth Never took it, 



Grade 


Grade 


Grade 


Not taking it 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 


O 



In either case, the questions take up much less than one page, allowing considerable room for 
instructions and for larger lists of courses . 5 

What are the difficulties and possible disadvantages of such a method? The most obvious 
difficulties, with commentary, are: 

How will transfer students respond? 

Transfer students would probably have to answer as best they could in terms of 
the course listing at their present school. The information would be less 
accurate, but it is better to have inaccurate data for a small group of identifiable 
students than for all students. 

Is a great deal more coding involved? 

Somewhat more coding would certainly be involved. Because the course page 
would not be uniform for all respondents, it might have to be coded in a separate 
process. In addition, to make maximum use of the course-taking information, a 
small amount of information from the schools would have to be coded as well 
(i.e., at least whether the course was one semester or two). On the positive side, 
a taxonomy of courses already exists (see the Transcript Study), and this would 
presumably be the basis for course coding. Also, one would not have to make 
separate coding decisions for every student— only for each school. 

Would this format take longer to fill out than the current format? 

This is a matter for testing. There may be more questions because there could be 
as many as a dozen or so individual courses. However, it would probably be 
easier for students to check off which of their school’s courses they have had than 
to try to decide how those courses fit into the categories used in the present 
questionnaires. 

Will this format require a great deal more preparation work? 

Somewhat. But the basic form would be the same throughout the country; only 
the specific course titles would vary. Gathering information on courses taught in 
a given school is straightforward (though one has to be careful to obtain 
information about courses over the past three years). 

Would there be a danger of mixing-up/losing forms? 

Certainly. But I believe that some form of identification is currently placed on 
the questionnaires; if so, it could be put on the separate course sheet as well. 



s lronically, there may be more mathematics courses within a given school than social studies courses. This 
occurs because of the multiple levels of certain mathematics courses. 



allowing later matching should they be separated accidentally or for purposes of 
coding. 

Would it be possible for an analyst to identify particular schools because of the 
particular combination of courses inquired about? 

If exact titles were coded and made available to analysts, this might be possible. 
However, if courses are coded according to a nationwide taxonomy, and if the 
original, school-specific forms are not made available, this should not be a 
problem. 

How would these questions be linked to questions about course content and methods? 
Linkages would have to be worked out on a subject-specific basis. As with the 
current procedure, one would have to pay careful attention to which courses were 
the subject of inquiry. For example, one might wish to obtain course-taking 
information about all social studies courses but information about course content 
only for history courses. This could be done through careful skip instructions— 
e.g., “if you did not fill in a circle for Q’s la and If (you have not had a U.S. 
history course), skip to Q. 10. 

Admittedly, the need to keep skip instructions simple could tax the skills of a good survey 
designer. It would also limit the number of types of courses one would wish to list; with a small 
number, skip instructions, if necessary, could be quite simple. In any event, I do not think this 
problem is insurmountable. Indeed, it might make distinctions between courses much clearer 
both for analysts and for students. Using the same illustrative courses as above (pp. 11-12), for 
example, one might re-order and label the courses as on the following page. This would make it 
possible to: a) ask both about enrollment in history and civics/govemment courses; and b) 
specify for students which courses they are to use as a basis for answering follow-up questions. 



Format 2 (revised): 



Questions la-lg. Have you taken or are you taking any of the following courses taught at 
NAME High School? Fill in one oval for each question. (If you transferred from another 
school, fill in circles for the courses that are closest to those named.) 

Ninth Tenth Eleventh Twelfth Never took it. 

Grade Grade Grade Grade Not taking it 



History courses 



la. American History 


O 


O 


O 


o 


o 


lb. A.P. American History 


O 


O 


O 


o 


o 


Civics/ American government courses 
lc. U.S. History and Government 


O 


O 


O 


o 


o 


Id. Participation in Government Internship 


O 


O 


O 


o 


o 


le. Participation in Government 


O 


O 


o 


o 


o 


If. Contemporary Issues 


O 


O 


o 


o 


o 


lg. A.P. U.S. Government & Politics 


O 


O 


o 


o 


o 



IF YOU DID NOT HAVE ANY OF THE CIVICS/ 
GOVERNMENT COURSES, SKIP TO Q. XX. 



(next page) 

2. In your civics/ American government courses, did you do any of the following....? 



Section (b) 



An alternative method of gathering course-taking information is to eschew student reports and 
collect the information directly from transcripts. I would not propose even considering such a 
method except that a large-scale transcript study is already done for NCES and is done largely 
with the same schools and students as are in the NAEP sample. Therefore, it is worth asking 
whether the two studies can be combined. 

For the most part, the advantages of such a procedure are obvious. In particular, one would have 
valid, detailed data about the courses taken by each student in a NAEP assessment, yet at the 
same time one could eliminate some of the questions on the subject questionnaires. 

While these advantages make the procedure worth at least brief consideration by those 
knowledgeable about the NAEP and Transcript Study procedures, I currently believe that 
administrative and other complications would make it unworkable. For one thing, if the 
transcript study is done only once every 3-5 years, as in the past, there will be some NAEP 
studies not associated with a corresponding transcript analysis. Increasing the frequency of the 
transcript study would raise, not lower, costs. The NAEP studies not associated with a transcript 
analysis would have to rely on other methods, such as the current ones, and the variation in 
methods would prevent any cost savings and raise additional validity problems. In addition, the 
willingness of schools to participate would have to be considered. While most schools that 
participate in one now participate in the other, that is not uniformly the case. Also, the Transcript 
Study is evidently done after graduation. A delay in getting information about the students might 
slow reporting of NAEP results, whereas speeding up the transcript analysis might raise other 
problems. Finally, it would still be necessary to gather information about course content and 
methods from students directly. Such questions constitute the bulk of the subject questionnaires, 
so the savings in the time taken to administer NAEP would be inconsequential. 



Problem 2: Level of Language/Understanding Expected of Fourth Graders 

Some of the vocabulary used on the 4th grade questionnaires appears to be too hard for the 
average student. I am not an expert in this field, but all of the inquires and simple analyses that I 
have made support this contention. 

The difficulty arises, first, in a specific item in the 1994 Geography Assessment and in the 
proposed 1998 Civics Assessment. The very first item asks 4th graders: 

"How often do you usually have social studies class in school?” 

(1994, Geography, Grade 4) 

“How often do you study social studies in school?” 

(1997 Field Test, Civics, Grade 4) 



I do not believe most 4th graders know what we mean by “social studies.” Moreover, even if at 
some level they understand the words, it is highly questionable whether we should rely on 
students’ interpretations of what is or is not included under this rubric. A similar question was 
asked on the 1994 History Assessment, but there it referred to “social studies or history,” so that 
4th graders may have had at least some basis for answering. 

More generally, consider a number of other words taken from various assessments: 

community, responsibilities, textbook, current events, filmstrips, panel discussions, role- 
playing, mock trial, dramas, opinion (1998 Civics pre-test) 

journals, regional concepts (1994 Geography) 

essays (1994 History) 

vocabulary, workbook (1994 Reading) 

contribute, collection, creativity, drafts, versions (1992 Writing) 

folder, portfolio (1994 Reading, Geography, History) 

None of these words are in the Edgar Dale’s “List of 3,000 Words Known by Students in Grade 
4,” 6 although that should only be taken as a preliminary indication that further consideration is 
needed. 

Even to the extent that individual words are understood, the phrasing does not seem conducive to 
understanding by all 4th graders. Surely 4th graders know whether the teacher cares “how much 
you’ve written,” but they are asked how important “the length of your paper” is to your teacher 
(1992 Writing, Q. 14). Similarly, 4th graders certainly understand the concept of “correcting 
your writing,” but they are asked about how often the teacher asks you to “write more than one 
draft of a paper” (1992 Writing, Q. 5). And while they would understand the concept if 
explained to them, one wonders what 4th graders think when asked whether they complete 
projects “that relate to the study of geography” (1994 Geography, Q. 6). 

There is also the matter of the accuracy of the 4th graders’ responses about classroom activities. 
So far, the matter seems to have been addressed only by noting that there are discrepancies 
between students’ and teachers’ responses, that the reason for these discrepancies is not clear, and 
that the reports represent their respective impressions of classroom activities ( 1994 U.S. History 
Report Card, p. 43). With older students, one might put some stock in students’ impressions. 



‘Jeanne S. Chall and Edgar Dale, Readability Revisited: The New Dale-Chall Readability Formula 
(Cambridge, MA: Brookline Books, 1995), pp. 16-29. 



With young students, one wonders whether they have much validity at all. I personally do not 
know of any study of the accuracy of young children’s responses (about any subject matter), but 
given the amount of error in adults’ responses, even in reporting factual matters such as visits to 
the dentist or the holding of library cards, 7 1 would anticipate very large amounts of error in 4th 
graders’ answers. 

Overall responses are only a partial guide in such matters, but consider these comparisons of 
teachers’ and students’ responses reported in the 1994 U.S. History Report Card (pp. 48-49): 

% write short answers % write reports almost every 
almost every day day/once or twice a week 

Teachers 9 6 

Students 35 24 

One suspects that students’ reports reveal not only “some discrepancies,” but a very considerable 
overestimate of the extent of writing that takes place in the classroom. For some purposes, such 
discrepant impressions might still be useful; as an assessment of what actually happens in the 
classroom (as would be judged by an independent observer, for example), they seem 
questionable at best. 



Solution: Simplify the Vocabulary and the Phrasing Used 

The most obvious solution is just to simplify the vocabulary and the phrasing used. This would 
require input from people who are familiar with 4th grade students and who are asked to pay 
special attention to the matter of vocabulary and “understandability.” 

Although I say “just” simplify the vocabulary and phrasing, this is not a trivial task. That is why 
I say that the questions need to be looked at by experts asked to pay special attention to these 
matters. I am aware that each of the questionnaires was constructed by committees that included 
specialists in elementary education. But if my experience on various Civics Assessment 
committees is any guide, the focus of attention was on the content of questions, and relatively 
little was said about readability level. 



’Overestimates of voter turnout and of voting for the winner have been well documented by political scientists 
(e.g., Brian D. Silver, et al. “Who Overreports Voting?” American Political Science Review (1986), 80:613-24). In 
the health area, see Ronald Anderson, et al., Total Survey Error (San Francisco: Jossey-Bass, 1979). More 
generally, see Seymour Sudman and Norman M. Bradbum, Asking Questions: A Practical Guide to Questionnaire 
Design (San Francisco: Jossey-Bass, 1986), pp. 56-63. 



A more radical solution would be to eliminate altogether the 4th grade subject-matter 
questionnaire. Doing so would shorten the amount of time required of 4th graders, easing the 
time pressures that plague NAEP. It might result in higher school cooperation rates (which I 
understand are a particular problem for fourth-grade civics). 

The tradeoff, of course, is that some information would be lost. My own assessment is that at 
least for the 4th grade, the loss would be minimal. I would be much more inclined to rely on 
information about classroom activities gathered from teachers than from students. 8 Even apart 
from the matter of question wording, I suspect that nine- and ten-year old children’s assessments 
of time and frequency are problematic and that even in the best of circumstances it is difficult to 
obtain accurate estimates from them about how often they do various activities. 9 Of course, for 
some purposes, we are interested only in relative estimates, not in absolute amounts of time (i.e., 
we only wish to know whether one set of students spends more time on a given activity than 
some other set of students). This eases the validity problem, but I would still prefer teachers’ 
reports to those of students. 

The objection may be raised, of course, that teachers are likely to bias their responses in a 
(natural) effort to make themselves “look good.” If so, one can treat these as “threatening 
questions” and use survey techniques suited to such questions. (See Sudman and Bradbum, 
Asking Questions, chap. 3). 

There is one item that cannot be obtained from teachers that would be a potentially important 
loss. That is a question about a student’s interest in the subject matter at hand (e.g., 1994 
Geography, grade 4, Q. 11: “How much do you like studying geography?”). If this question, but 
not others, were deemed vitally important, it could possibly be attached to the end of the test 
instrument (as are questions about amount of effort on the Assessment). 



Field-Testing of New Methods 

Many of the changes I have suggested involve only small revisions of the subject questionnaires. 
While even modest changes should be pre-tested, they do not represent fundamentally new 
procedures that need elaborate testing. 

However, the method suggested in Part 5 of listing school-specific courses is a more radical 
change and deserves more serious testing. Likewise, the problems associated with the level of 
language used in the fourth-grade questionnaires deserve careful testing of proposed revisions. 



“Personally, I would be inclined to go further and to rely on teacher information about 8th and possibly even 
12th grade courses. 

’Certainly adults have such difficulties (Sudman and Bradbum, Asking Questions, pp. 36-51). 



To begin, I would point out that the question about Advanced Placement courses that has been 
field-tested for the 1998 Civics Assessment provides an initial test of a course-specific approach. 
If this question works out well, it will lend support to such a method. 

Testing this approach more fully should not be difficult. Indeed, I suspect that the greatest 
difficulty— and one of the main benefits— lies in the need to think more carefully than heretofore 
about exactly what course information is wanted. In any event, tests are necessary both of 
feasibility and of the quality of the data collected. By feasibility, I am referring to the ability and 
willingness of students to respond to questions of the type suggested. My belief is that students 
will be better able to answer course-specific questions precisely because they know exactly what 
they are being asked. Nonetheless, it is possible that students will balk at trying to remember 
exactly what courses they had two and three years earlier. Providing course names should make 
the task easy, but there is no substitute for testing. 10 

To test results for quality, one might compare the responses to known information about a 
school’s courses. For example, one can see how many students say that they took U.S. history in 
1 Oth grade when it is known that virtually all students take it in 9th grade. A better test would be 
to compare the responses to student transcripts. For a small study, this would not be overly time 
consuming or costly. 

With respect to the level of language in the 4th grade questionnaires, the use of focus groups 
might be worthwhile. Focus group discussions would generate considerable insight into the way 
in which young students interpreted the questions and the kind of information and perspectives 
they used in answering them. Discussions with teachers— and perhaps with teachers and students 
together— would help provide information on the quality of students’ answers to questions about 
course content and methods. Indeed, if successful, it might be useful to expand the use of focus 
groups to 8th graders and to high school students to gain insight into their responses and, 
ultimately, to help design better questionnaires. 



I0 I have occasionally asked college students what courses they are taking now and received a blank stare while 
they slowly dredged up the information. 



Attachment 1 



Percentage of 12 Graders Reporting That They Have Taken an 
Advanced Placement Course by Self-Reported Grade and by 
High School Program (1988) 



Grade/program 


% AP course 


N 


Mostly A 


18.5 


790 


Half A, half B 


16.7 


669 


Mostly B, half B/half C 


15.1 


1441 


Mostly C, half C/half D 


12.1 


697 


Mostly D, mostly below D 


17.9 


151 


Academic/college 


16.3 


2389 


General 


13.3 


1476 


Vocational/technical 


13.6 


389 



Note: Unweighted N’s, 12th graders only, 1988 NAEP (Civics). 
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